It's August Bank Holiday Weekend, we're in Cambridge. It must be
the Debian
UK OMGWTFBBQ!.
We're about halfway through, and we've already polished off lots
and lots of good food and beer. Lars is making pancakes as I write
this, :-) We had an awesome game of Mao last night. People are having
fun!
Many thanks to a number of awesome friendly people for again
sponsoring the important refreshments for the weekend. It's
hungry/thirsty work celebrating like this!
Back in June 2018, Michael Stapelberg put the Raspberry Pi image building up
for
adoption. He
created the first set of unofficial, experimental Raspberry Pi images for
Debian. I promptly answered to him, and while it took me some time to actually
warp my head around Michael s work, managed to eventually do so. By December, I
started pushing some
updates.
Not only that: I didn t think much about it in the beginning, as the needed
non-free pacakge was called raspi3-firmware, but By early 2019, I had it
running for all of the then-available Raspberry families (so the package was
naturally renamed to
raspi-firmware). I got my
Raspberry Pi 4 at DebConf19 (thanks to Andy, who brought it from Cambridge), and
it soon joined the happy Debian family. The images are built daily, and are
available in https://raspi.debian.net.
In the process, I also adopted Lars great vmdb2 image building
tool, and have kept it decently up to
date (yes, I m currently lagging behind, but I ll get to it soonish ).
Anyway This year, I have been seriously neglecting the Raspberry builds. I
have simply not had time to regularly test built images, nor to debug why the
builder has not picked up building for trixie (testing). And my time
availability is not going to improve any time soon.
We are close to one month away from moving for six months to Paran
(Argentina),
where I ll be focusing on my PhD. And while I do contemplate taking my
Raspberries along, I do not forsee being able to put much energy to them.
So This is basically a call for adoption for the Raspberry Debian images
building service. I do intend to stick around and try to help. It s not only me
(although I m responsible for the build itself) we have a nice and healthy
group of Debian people hanging out in the #debian-raspberrypi channel in OFTC
IRC.
Don t be afraid, and come ask. I hope giving this project in adoption will
breathe new life into it!
KDE Mascot/etc/network/interfaces:
allow-hotplug eno1
iface eno1 inet dhcp
pre-up iptables-restore /etc/network/iptables.up.rules
iface eno1 inet6 dhcp
pre-up ip6tables-restore /etc/network/ip6tables.up.rules
iptables dispatcher script for
NetworkManager,
I decided to follow the same approach for systemd-networkd.
I started by installing networkd-dispatcher:
apt install networkd-dispatcher
/etc/networkd-dispatcher/routable.d/iptables:
#!/bin/sh
LOGFILE=/var/log/iptables.log
if [ "$IFACE" = lo ]; then
echo "$0: ignoring $IFACE for \ $STATE'" >> $LOGFILE
exit 0
fi
case "$STATE" in
routable)
echo "$0: restoring iptables rules for $IFACE" >> $LOGFILE
/sbin/iptables-restore /etc/network/iptables.up.rules >> $LOGFILE 2>&1
/sbin/ip6tables-restore /etc/network/ip6tables.up.rules >> $LOGFILE 2>&1
;;
*)
echo "$0: nothing to do with $IFACE for \ $STATE'" >> $LOGFILE
;;
esac
chmod a+x /etc/NetworkManager/dispatcher.d/pre-up.d/iptables
/etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules) and
use the handy iptables-apply and ip6tables-apply commands to test
any changes to my firewall rules.
Looking at /var/log/iptables.log confirms that it is being called
correctly for each network interface as they are started.
IZ #294, the latest issue
IZ #194: The first by TTA press
Thanks to Nicholas Guriev,
dotenv-cli now uses exec instead of popen to create the new process on
POSIX systems.
As a refresher, dotenv-cli is a package that
provides the dotenv command. dotenv reads the .env file from the current
directory, puts the contents in the environment variables, and executes the
given command with the extra environment variables set.
dotenv comes in handy if you follow the 12 factor app
methodology or just need to run a program locally with specific environment
variables set.
With this new change, when you call
dotenv my_awesome_tool
my_awesome_tool, effectively creating a
child process of dotenv, dotenv now uses exec to become the new
process. This is a bit cleaner, as there is no longer a dotenv-process
running that you don t actually care about, and less-error prone when trying to
send signals such as SIGTERM to the processes that runs my_awesome_tool.
Unfortunately, exec does not work properly under Windows, so here we still
fall back to using popen.
This new feature is available in version 3.2.0 which is available on PyPI
and debian/unstable.
co2mon.nz currently uses monitors based on Oliver Seiler s open source design which I am personally building. This post describes my exploration of how to achieve production of a CO2 monitor that could enable the growth of co2mon.nz.
I started out trying to use the EasyEDA/OSHWLab ecosystem thinking the tight integration with JLCPCB s assembly services would be a benefit, but the web interface was too clunky and limiting and I quickly got frustrated. KiCad proved to be a much more pleasant and capable tool for the job.
The reference design in the ESP32 datasheet (p28) and USB-C power supply examples from blnlabs were particularly helpful alongside the KiCad documentation and the example of the existing monitor in completing this step (click the image to enlarge).
Taking all of these factors into account I ended up with a square PCB containing a cutout in the top right so that the ESP32 antenna can sit within the overall square outline while still meeting its design requirements. The SCD40 and BME680 sit in the top left corner, near the edges for good airflow and far away from the SHT30 temperature sensor in the bottom left corner. The LEDs I placed in a horizontal row across the center of the board, the LCD in the bottom right, a push button on the right-hand side and the USB-C socket in the center at the bottom.
Once the components are placed, the next big task is to route the traces (aka wires) between the components on the board such that all the required electrical connections are made without any unintended connections (aka shorts) being created. This is a fun constraint solving/optimisation challenge and takes on an almost artistic aspect with other PCB designers often having strong opinions on which layout is best. The majority of the traces and routing for this board were able to be placed on the top layer of the PCB, but I also made use of the back layer for a few traces to help avoid conflicts and deal with places where different traces needed to cross each other. It s easy to see how this step would be much more challenging and time consuming on a larger and more complex PCB design.
The final touches were to add some debugging breakouts for the serial and JTAG ports on the ESP32-S3 and a logo and various other helpful text on the silkscreen layer that will be printed on the PCB so it looks nice.
The final surprise in the assembly process was the concept of edge rails, additional PCB material that is needed on either side of the board to help with feeding it through the assembly machine in the correct position. These can be added automatically by JLCPCB and have to be snapped off after the completed boards are received. I hadn t heard about these before and I was a little worried that they d interfere or get in the way of either the antenna cut-out at the top of the board, or the switch on the right hand side as it overhangs the edge so it can sit flush with the case.
In the end there was no issue with the edge rails. The switch was placed hanging over them without issue and snapping them off once the boards arrived was a trivial 30s job using a vice to hold the edge rail and then gently tipping the board over until it snapped off - the interface between the board and the rails while solid looking has obviously been scored or perforated in some way during the production process so the edge breaks cleanly and smoothly. Magic!
The process was amazingly quick with the completed PCBs (picture above) arriving within 7 days of the order being placed and looking amazing.
As a result I ended up completely flipping the design such that the front panel is a single piece of plastic that also encompasses the walls of the case and contains appropriate mounting stakes for both the screen and the main PCB.
Getting to this design hugely simplified the assembly process. Starting with an empty case lying face down on a bench, the LCD screen is pushed onto the mounting poles and sits flush with the cover of the case - easily achieved without the main PCB yet in place.
Next, the main PCB is gently lowered into the case facing downwards and sits on the mounting pole in each corner with the pins for the LCD just protruding through the appropriate holes in the PCB ready to be quickly soldered into place (this took significant iteration and tuning of dimensions/positioning to achieve!).
Finally, a back panel can be attached which holds the PCB in place and uses cantilever snap joints to click on to the rest of the case.
Overall the design is a huge improvement over the previous case which required screws and spacers to position the PCB and cover relative to the rest of the case, with the spacers and screws being particularly fiddly to work with.
The major concern I had with the new design was that the mount to attach the monitor to the wall has moved from being attached to the main case and components directly to needing to be on the removable back panel - if the clips holding this panel to the case fail the core part of the monitor will fall off the wall which would not be good. To guard against this I ve doubled the size and number of clips at the top of the case (which bears the weight) and the result seems very robust in my testing. To completely assemble a monitor, including the soldering step takes me about 2-3 minutes individually, and would be even quicker if working in batches.
I tried a variety of filament colours, but settled on a transparent filament which once combined in the necessary layers to form the case is not actually transparent like perspex is, but provides a nice translucent medium which achieves the goal of having the light colour visible without exposing all of the circuit board detail. There s room for future improvement in the positioning of the LEDs on the circuit board to provide a more even distribution of light across the case but overall I really like the way the completed monitor ends up looking.
The goal of having a CO2 monitor which I can outsource the vast majority of production of is as close to being met as I think is possible without undertaking the final proof of placing a large order. I ve satisfied myself that each step is feasible and that the final assembly process is quick, easy and well below the level of effort and time it was taking me to produce the original monitors.
Cost wise it s also a huge win, primarily in terms of the time taken, but also in the raw components - currently the five prototypes I ordered and built are on par with the component cost of the original CO2 monitor, but this will drop further with larger orders due to price breaks and amortisation of the setup and shipping expenses across more monitors.
This project has also given me a much better appreciation for how much I m only just scratching the surface of the potential complexities and challenges in producing a hardware product of this type.
I m reasonably confident I could successfully produce a few hundred and maybe even a few thousand monitors using this approach, but it s also clear that getting beyond that point is and would be a whole further level of effort and learning.
Hardware is hard work. That s not news to anyone, including me, but there is something to be said for experiencing the process first hand to make the reality of what s required real.
The PCB and case designs are both shared and can be found at https://github.com/co2monnz/co2monitor-pcb and https://github.com/co2monnz/cad, feedback and suggestions welcome!
Version 0.0.13 of RcppSpdlog is now
on CRAN and will be soon be
uploaded to Debian too. RcppSpdlog
bundles spdlog, a
wonderful header-only C++ logging library with all the bells and
whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn
more at the package
documention site.
This release adds a small (but handy) accessor generalisation:
Instead of calling setup() with two arguments for a label
and the logging level we now only require the desired level. We also
cleaned up one implementation detail for the stopwatch
feature added in January, and simplified the default C++ compilation
standard setting.
The NEWS entry for this release follows.
Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in RcppSpdlog version 0.0.13 (2023-06-17)
- Minor tweak to
stopwatchsetup avoids pulling in fmt- No longer set a C++ compilation standard as the default choices by R are sufficient for the package
- Add convenience wrapper
log_initomitting first argument tolog_setupwhile preserving the interface from the latter- Add convenience
setupwrappersinitandlogto API header filespdl.h
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
cpanm Mail::Milter::Authentication.
Wah. I m glad I tried this out on a test virtual machine. It took forever! It
ran tests! It compiled things! And, it installed a bunch of perl modules
already packaged in Debian.
I don t think I want to add this command to my ansible playbook.
Next I spent an inordinate amount of time trying to figure out how to list the
dependencies of a given CPAN module. I was looking for something like cpanm --list-dependencies Mail::Milter::Authentication but eventually ended up writing
a perl script that output
perl code, inserting a use " before each dependency and a semicolon and line
break after them. Then, I could execute that script on a clean debian
installation and see which perl modules I needed. For each error, I checked for
the module in Debian (and installed it) or kept a list of modules I would have
to build (and commented out the line).
Once I had a list of modules to build, I used the handy cpan2deb command. It
took some creative ordering but eventually I got it right. Since I will surely
forget how to do this when it s time to upgrade, I wrote a
script.
In total it took me several days to figure this all out, so I once again find
myself very appreciative of all the debian packagers out there - particularly
the perl ones!!
And also if I did this all wrong and there is an easier way I would love to
hear about it in the comments.
I wrote the Ruby bindings for the Enquo Project, my attempt to bring queryable encryption to all databases, using the Rutie library.
Recently, I ve rewritten the bindings to use Magnus instead, and I thought I d put down my thoughts about the whole situation.
Wrong sort of shim, but canned language bindings would be handy
u32 type, which can only store integers between zero and about four billion (232 - 1, to be precise).
There s also lots of little things that need to be just right, also, like translating the different memory management approaches of the languages, and dealing with a myriad of fiddly little issues like passing arguments and return values in and out of method calls, helpers for defining classes and methods (and pointing to the correct Rust functions), and so on.
This is what I imagine it looks like inside these libraries
self reference to the current object):
fn enquo_field_decrypt_text(ciphertext_obj: RString, context_obj: RString) -> RString
let ciphertext = ciphertext_obj.to_str_unchecked();
let context = context_obj.to_vec_u8_unchecked();
let field = rbself.get_data(&*FIELD_WRAPPER);
// etc etc etc
fn decrypt_text(&self, ciphertext: String, context: String) -> Result<String, magnus::Error>
Result return value, while Rutie s approach to raising an exception involves poking the Ruby VM directly, which always struck me as a bit ugly.
There are several other minor things in Magnus (like its cleaner approach to wrapping structs so they can be stored in Ruby objects) that I m appreciating, too.
Never discount the power of ergonomics for making a happy developer.
$ git diff --stat -- lib ext/enquo/src
ruby/ext/enquo/src/field.rs 342 ++++++++++++++++++++++++++++++++++++++
ruby/ext/enquo/src/lib.rs 338 ++++---------------------------------
ruby/ext/enquo/src/root.rs 39 +++++
ruby/ext/enquo/src/root_key.rs 67 ++++++++
ruby/lib/enquo.rb 6 +-
ruby/lib/enquo/field.rb 173 -------------------
ruby/lib/enquo/root.rb 28 ----
ruby/lib/enquo/root_key.rb 1 -
ruby/lib/enquo/root_key/static.rb 27 ---
9 files changed, 479 insertions(+), 542 deletions(-)
raise ArgumentError if something.isnt_right code in those .rb files.
So, in conclusion, if you, too, are building Ruby extensions in Rust, while Rutie is a solid choice (and you probably should stick with it if you re already using it), I highly recommend giving Magnus a look for your next extension.
Just some of the organisations that leaked data in 2022
Not very useful for a database
Indeed
Everyone who uses a database...
username: JABBER-ID password: PASSWORDGo-sendxmpp can take a username and password on the command-line but that s bad for security as in the absence of SE Linux or other advanced security systems the password can be seen by any user on the same system who runs ps. To send a message run echo $MESSAGE go-sendxmpp $ADDR to send $MESSAGE to $ADDR. It also has the option go-sendxmpp -l to listen for incoming messages. I don t have an immediate need to receive messages from the command-line but it s handy to have the option. I probably won t be able to get a new version of etbemon in Debian for the Bookworm release. So to get go-sendxmpp to work with etbemon you need to edit /usr/lib/mon/alert.d/mailxmpp.alert and change this sendxmpp line to this go-sendxmpp line:
open (XMPP, " /usr/bin/sendxmpp -a /etc/ssl/certs -t @xmpprec -r $host") open (XMPP, " /usr/bin/go-sendxmpp @xmpprec")
netstat to sockstat transition.
I used to do this to show which processes where listening on which
port on a server:
netstat -anpe
-a), not resolve hostnames (-n, because it's slow), show
processes attached to the socket (-p) with extra info like the user
(-e). This still works, but sometimes fail to find the actual
process hooked to the port. Plus, it lists a whole bunch of UNIX
sockets and non-listening sockets, which are generally irrelevant
for such an audit.
What I really wanted to use was really something like:
netstat -pleunt sort
-l) and network sockets, specifically UDP (-u) and TCP
(-t).
But enough with the legacy, let's try the brave new world of sockstat
which has the unfortunate acronym ss.
The equivalent sockstat command to the above is:
ss -pleuntO
-O flag otherwise ss
does that confusing thing where it splits the output on multiple
lines. But I actually use:
ss -plunt0
-e as the information it gives (cgroup, fd
number, etc) is not much more useful than what's already provided with
-p (service and UID).
All of the above also show sockets that are not actually a concern
because they only listen on localhost. Those one should be filtered
out. So now we embark into that wild filtering ride.
This is going to list all open sockets and show the port number and
service:
ss -pluntO --no-header sed 's/^\([a-z]*\) *[A-Z]* *[0-9]* [0-9]* *[0-9]* */\1/' sed 's/^[^:]*:\(:\]:\)\?//;s/\([0-9]*\) *[^ ]*/\1\t/;s/,fd=[0-9]*//' sort -gu
anarcat@angela:~$ sudo ss -pluntO --no-header sed 's/^\([a-z]*\) *[A-Z]* *[0-9]* [0-9]* *[0-9]* */\1/' sed 's/^[^:]*:\(:\]:\)\?//;s/\([0-9]*\) *[^ ]*/\1\t/;s/,fd=[0-9]*//' sort -gu
[::]:* users:(("unbound",pid=1864))
22 users:(("sshd",pid=1830))
25 users:(("master",pid=3150))
53 users:(("unbound",pid=1864))
323 users:(("chronyd",pid=1876))
500 users:(("charon",pid=2817))
631 users:(("cups-browsed",pid=2744))
2628 users:(("dictd",pid=2825))
4001 users:(("emacs",pid=3578))
4500 users:(("charon",pid=2817))
5353 users:(("avahi-daemon",pid=1423))
6600 users:(("systemd",pid=3461))
8384 users:(("syncthing",pid=232169))
9050 users:(("tor",pid=2857))
21027 users:(("syncthing",pid=232169))
22000 users:(("syncthing",pid=232169))
33231 users:(("syncthing",pid=232169))
34953 users:(("syncthing",pid=232169))
35770 users:(("syncthing",pid=232169))
44944 users:(("syncthing",pid=232169))
47337 users:(("syncthing",pid=232169))
48903 users:(("mosh-client",pid=234126))
52774 users:(("syncthing",pid=232169))
52938 users:(("avahi-daemon",pid=1423))
54029 users:(("avahi-daemon",pid=1423))
anarcat@angela:~$
ss -pluntO --no-header \
sed 's/^\([a-z]*\) *[A-Z]* *[0-9]* [0-9]* *[0-9]* */\1/;s/^tcp//;s/^udp//' \
grep -v -e '^\[fe80::' -e '^127.0.0.1' -e '^\[::1\]' -e '^192\.' -e '^172\.' \
sed 's/^[^:]*:\(:\]:\)\?//;s/\([0-9]*\) *[^ ]*/\1\t/;s/,fd=[0-9]*//' \
sort -gu
anarcat@angela:~$ sudo ss -pluntO --no-header sed 's/^\([a-
z]*\) *[A-Z]* *[0-9]* [0-9]* *[0-9]* */\1/;s/^tcp//;s/^udp//'
grep -v -e '^\[fe80::' -e '^127.0.0.1' -e '^\[::1\]' -e '^192\.' -
e '^172\.' sed 's/^[^:]*:\(:\]:\)\?//;s/\([0-9]*\) *[^ ]*/\
1\t/;s/,fd=[0-9]*//' sort -gu
22 users:(("sshd",pid=1830))
500 users:(("charon",pid=2817))
631 users:(("cups-browsed",pid=2744))
4500 users:(("charon",pid=2817))
5353 users:(("avahi-daemon",pid=1423))
6600 users:(("systemd",pid=3461))
21027 users:(("syncthing",pid=232169))
22000 users:(("syncthing",pid=232169))
34953 users:(("syncthing",pid=232169))
35770 users:(("syncthing",pid=232169))
48903 users:(("mosh-client",pid=234126))
52938 users:(("avahi-daemon",pid=1423))
54029 users:(("avahi-daemon",pid=1423))
lsof can do
some of this, and it's relatively straightforward. This lists all
listening TCP sockets:
lsof -iTCP -sTCP:LISTEN +c 15 grep -v localhost sort
lsof -i @localhost
grep -v localhost line.
In theory, this would do the equivalent on UDP
lsof -iUDP -sUDP:^Idle
lsof: no UDP state names available: UDP:^Idle
ss can
figure out the state of those sockets, heck it's how -l vs -a
works after all. So we need something else to show listening UDP
sockets.
The following actually looks pretty good after all:
ss -pluO
localhost sockets of course, so we can explicitly ask
ss to resolve those and filter them out with something like:
ss -plurO grep -v localhost
ss supports pattern matching, so we can actually
tell it to ignore localhost directly, which removes that horrible
sed line we used earlier:
ss -pluntO '! ( src = localhost )'
ss -nplutO '! ( src = localhost )' \
sed 's/\(udp\ tcp\).*:\([0-9][0-9]*\)/\2\t\1\t/;s/\([0-9][0-9]*\t[udtcp]*\t\)[^u]*users:(("/\1/;s/".*//;s/.*Address:Port.*/Netid\tPort\tProcess/' \
sort -nu
anarcat@angela:~$ sudo ss -nplutO '! ( src = localhost )' sed 's/\(udp\ tcp\).*:\([0-9][0-9]*\)/\2\t\1\t/;s/\([0-9][0-9]*\t[udtcp]*\t\)[^u]*users:(("/\1/;s/".*//;s/.*Address:Port.*/Port\tNetid\tProcess/' sort -nu
Port Netid Process
22 tcp sshd
500 udp charon
546 udp NetworkManager
631 udp cups-browsed
4500 udp charon
5353 udp avahi-daemon
6600 tcp systemd
21027 udp syncthing
22000 udp syncthing
34953 udp syncthing
35770 udp syncthing
48903 udp mosh-client
52938 udp avahi-daemon
54029 udp avahi-daemon
| Publisher: | Amazon Original Stories |
| Copyright: | September 2019 |
| ISBN: | 1-5420-9206-X |
| ISBN: | 1-5420-4363-8 |
| ISBN: | 1-5420-9357-0 |
| ISBN: | 1-5420-0434-9 |
| ISBN: | 1-5420-4363-8 |
| ISBN: | 1-5420-4425-1 |
| Format: | Kindle |
| Pages: | 300 |
typing.overload
typing.overload
makes it easier to type functions with behaviour that depends on input types.
Functions marked with @overload are ignored by Python and only used by the
type checker:
@overload
def process(response: None) -> None:
...
@overload
def process(response: int) -> tuple[int, str]:
...
@overload
def process(response: bytes) -> str:
...
def process(response):
# <actual implementation>
set_start_method("spawn"), but when we
tried it we hit serious performance penalties.
Lesson learnt: multiprocessing is good for prototypes, and may end up being too
hacky for production.
In my case, I was already generating small python scripts corresponding to
worker tasks, which were useful for reproducing and debugging
Magics issues, so I
switched to running those as the actual workers. In the future, this may come
in handy for dispatching work to HPC nodes, too.
Here's a parallel execution scheduler based on asyncio
that I wrote to run them, which may always come in handy on other projects.
| Series: | Innkeeper Chronicles #6 |
| Publisher: | NYLA Publishing |
| Copyright: | 2022 |
| ISBN: | 1-64197-239-4 |
| Format: | Kindle |
| Pages: | 440 |
Version 0.0.11 of RcppSpdlog is now on CRAN and in Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich.
This release adds support for a basic file logger as a alternative to the console logger. This can be helpful with code which suppresses or hides console output as for example unit test code does. We also expose the formatting helper function for direct use at the C level from other packages, and mention the handy wrapper spdl in the README.
The NEWS entry for this release follows.
Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in RcppSpdlog version 0.0.11 (2022-12-13)
- Export the formatter at C level
- Mention package spdl in README.md
- Support simple file-based logger
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Next.